The (mysterious) case of text data
Unlocking the power of text data
Example: inaugural speeches of presidents
“Fellow citizens, I am again called upon by the voice of my country to execute the functions of its Chief Magistrate. When the occasion proper for it shall arrive, I shall endeavor to express the high sense I entertain of this distinguished honor, and of the confidence which has been reposed in me by the people of united America. […]”
Going beyond summary variables…
N-grams
## people government us can upon must great
## 575 564 478 471 371 366 340
## may states shall
## 338 333 314
## of_the in_the to_the of_our people government us
## 1766 812 720 619 575 564 478
## can and_the upon
## 471 471 371
| name | ngrams |
|---|---|
| Democratic | us, people, can, government, must, nation, new, world, shall, every |
| Republican | people, government, can, us, must, upon, world, great, country, peace |
| name | ngrams |
|---|---|
| Obama | us, must, can, people, nation, new, time, every, america, now |
| Trump | america, american, people, country, one, every, never, great, nation, new |
| document | rights | justice | important | revolution | individual | system | many | |
|---|---|---|---|---|---|---|---|---|
| 53 | 1997 | 1 | 1 | 1 | 2 | 0 | 0 | 0 |
| 54 | 2001 | 1 | 3 | 2 | 0 | 0 | 0 | 5 |
| 55 | 2005 | 5 | 6 | 0 | 0 | 0 | 0 | 1 |
| 56 | 2009 | 1 | 0 | 0 | 1 | 1 | 1 | 3 |
| 57 | 2013 | 2 | 2 | 0 | 0 | 2 | 0 | 1 |
| 58 | 2017 | 0 | 1 | 0 | 0 | 0 | 1 | 5 |
Suppose we have got two vectors:
Uses Pythagorean theorem.
For two 2-dimensional locations:
## Maths with text
Obama’s speeches were more similar to one another
| Year | Trump |
|---|---|
| 2017 | 0.00000 |
| 2001 | 43.26662 |
| 1977 | 48.27007 |
| 1945 | 50.07994 |
| 1905 | 51.34199 |
| 1793 | 51.46844 |
These are: Bush (R), Carter (D), Roosevelt (D)
| Year | Trump |
|---|---|
| 1841 | 290.5013 |
| 1845 | 153.2057 |
| 1909 | 148.8456 |
| 1897 | 123.7901 |
| 1821 | 122.0819 |
| 1889 | 116.2196 |
These are: Harrison, Polk, Taft, McKinley, Monroe, Harrison
What’s more?
Thanks.